Load-balance, active-box windowing, block-structured AMR - #1628

Step 4 of docs/documentation/amr_per_level_distribution.md. amr_fine_cut becomes one independent SFC cut per level, so every level is balanced across all ranks on its own weight; the twt tower rollup, the descendant-inherit pass, and f_amr_owner's level-1 anchor walk are deleted. A deep refinement tower no longer pins its whole subtree (weight cost*rr**(l*d)) to one rank, which was the granularity cap at depth.

Per level rather than one mixed cut over all fine blocks: same-level boxes are disjoint and so carry distinct Morton keys, which the cut-point binary search needs. Mixed, a level-2 block sharing its parent's region_lo would collide with it.

Splitting towers made four latent reads reachable, all of the same shape - a parent-slot field guarded on owning the CHILD, safe only while the two owners always coincided. s_set_amr_fine_geometry read amr_slots(pblk)%amr_ref_ratio (written by s_amr_alloc_slot, never called for pblk on a rank that does not own it, so undefined) and bisected amr_slots(pblk)%x_cb (unallocated there); the ghost-shell extension took lbound/ubound of the same unallocated array. Together they produced garbage cell widths and NaNs at the rank seam a few steps in - EF58E377, the one golden whose level-2 tiles actually divide across ranks.

Footprint and refinement ratio now come from s_amr_parent_foot, and coordinates from a new s_amr_build_block_coords that replays the ancestor chain from the global L0 boundaries. Refinement is nested midpoint subdivision, so the replay reproduces exactly what bisecting the parent's stored array gave, and it collapses the level-1 and level>=2 coordinate branches into one call. The subcycle SETUP gather gets the same participant split the runtime gather already had.

Precheck 7/7, 70/70 AMR goldens - the first run with a genuinely split tower, so the cross-rank gather/restrict/reflux are exercised rather than dormant.

…asured outcome

The plan doc named load balancing once, only to reject an alternative - it had no mechanism, no metric, and no acceptance criteria, while the goal of the work is AMR *plus* load balancing.

Adds a Load balancing section: the three mechanisms often conflated (L0 Cartesian rebalance, the cost model / imbalance diagnostic, and the AMR block assignment), the max/mean per-rank-weight metric with scale invariance as the requirement that actually matters, and the four structural limits that bind at scale - granularity floor, indivisible atom, staleness within a regrid interval, and cost-model blindness when load_weight_wrt is off. Notes that step 4 is the load-balancing enabler rather than only a distribution change: no assignment can fix an imbalance whose atom is a whole tower.

Records the step-4 A/B honestly. Per-level distribution measured FLAT against co-location at np=1/2/4/8, and the result is not attributable because there is no imbalance metric - a flat end-to-end time cannot separate 'nothing to fix' from 'balancer did nothing'. Instrumentation therefore becomes step 5 and precedes further distribution work. Also records that the same case costs ~20x the coarse solve while its refinement accounts for ~2.5x, putting the dominant term in per-block launch overhead rather than balance.

Marks steps 1-4 landed with commits, rewrites the stale Track 2 section, adds the parent-slot-guarded-on-the-child bug class that splitting towers exposed, and corrects the validation bar: cross-compiler coverage is CI's job, the local gate is amdflang OMP offload.

…s inconclusive

The uniform control reaches only 18% parallel efficiency at np=8 (262k cells/rank, below what saturates an MI250X GCD), so both arms are latency-bound exactly where per-level distribution should matter most. The flat A/B therefore could not have detected the effect and is inconclusive, not negative. Adds the efficiency table and a standing rule: read the control's efficiency before trusting anything measured against it.

Also corrects the overhead figure. It was quoted at np=8, where the uniform number is already degraded; at np=1, where both arms are compute-bound, AMR costs 40.7x the coarse solve for ~2.5x the nominal work - a ~16x residual, not ~8x.

The step-4 A/B measured flat at every rank count and could not be interpreted: with no imbalance metric a flat time cannot separate 'no imbalance to fix' from 'balancer did nothing' from 'gain offset by new P2P traffic'. Print the quantity the SFC cut is actually minimising - max_r W(r) / mean_r W(r) per level and in total - plus the count of ranks assigned nothing, which is the granularity floor showing up directly. Needs no MPI: wt, amr_block_level and amr_block_owner are replicated and identical on every rank (the cost vector is allreduced in s_amr_block_cost), so rank 0 prints. Gated behind load_weight_wrt, so every golden is unaffected. Precheck 7/7, 70/70 AMR goldens.

Audit of fc53e09 found two flaws in the report itself. (1) merge(mx/mean, 1._wp, mean > 0._wp) does not guard: merge is a function, so both arms are evaluated and the mean == 0 arm still divides by zero. Use a real if. (2) rw/tw were automatic arrays dimensioned 0:num_procs-1 - O(num_procs) on the STACK, in the one routine whose purpose is measuring behavior at large rank counts; the module already heap-allocates amr_owner_cut and amr_fine_cut at exactly that size. Allocate them. Also rename idle_ranks -> no_blocks_ranks / ranks_with_no_fine_block: the counter is per-LEVEL, and a rank with no block at a level still owns level-0 work, so the old name invited reading it as idleness. It is not; only m_rank_timing measures that. Separately, the s_amr_advance_fine_subcycle_all docstring claimed the L2-L2 seam halo was future work and that s_amr_check_seam_topology aborts on L2+ seams under subcycle - 3db24df removed that abort and the halo exists; and the s_amr_tile_box tsz docstring named a fixed amr_maxc_fit/2. Precheck 7/7, 71/71 AMR goldens.

The regrid child-box cap ran inside do lev = 2, amr_max_level but passed amr_maxc_fit/2 at every level. That constant is derived for level 2 specifically: a slot holds amr_ref_ratio*amr_maxc_fit fine cells (max_f1 = amr_ref_ratio*amr_maxc_fit - 1) and a level-lev block spans amr_ref_ratio**lev fine cells per coarse cell, so the cap must be amr_maxc_fit/amr_ref_ratio**(lev-1). At lev = 3 the fixed /2 admits a box twice what the slot holds. Identical at lev = 2 (amr_ref_ratio is 2 whenever amr_max_level > 1 - the checker gates rr /= 2 to single-level), so no existing golden moves, and 71/71 AMR goldens confirm that. NOT VERIFIED BY RUNNING: no golden covers amr_max_level >= 3, so this is arithmetic checked against the slot sizing, not observed behavior. It is strictly more correct than the constant it replaces and cannot regress level 2, but do not read the green suite as evidence that level 3 works. No call-site floor is needed - s_amr_tile_box already does tc = max(tc, 1) for exactly the zero case (collapsed dims pass 0 today).

The PROHIBIT's stated reason - that s_amr_advance_children advances level-2 children per-block with no L2-L2 halo - stopped being true at 3db24df, which removed the matching runtime abort saying 'the halo it warned about now exists'. 26e0d08 then extended the transpose across parents. Blast radius is narrow: every risky combination keeps its own gate (non-polytropic QBMM np>1, cyl_coord, IB np>1, moving IB, amr_ref_ratio /= 2, amr_max_blocks < 2), so this opens only plain Cartesian hydro at rr=2. No test expected the abort. New golden C45DBB52 covers it, VERIFIED by dumping block lo/hi/owner rather than assumed: adjacent L2 siblings [7,11]+[12,16] under one parent, and a child on a different rank from its parent (L2 [52,56] rank 0 under L1 [49,59] rank 1) at an intermediate regrid - the per-substep P2P SETUP gather, whose failure mode is a deadlock no tolerance check catches. NOT covered, deliberately: the L2 seam over MPI. Morton order keeps a contiguous run contiguous and the SFC cut is a single split point, so the adjacent pair lands on one rank; an eps x buf sweep (0.002-0.05 x 8,12) never split it, and contriving one would pin the golden to an SFC outcome any cost-model change would undo. Both ingredients are covered separately (level-2 fmul by A635AA56 at np=1, MPI seam transport by the single-level subcycle np=2 golden). Note cross-parent seams are impossible at any depth: the nesting window insets each parent by amr_cpat_mar >= 1. Precheck 7/7, 71/71 AMR goldens.

The granularity-floor section said a shallow hierarchy leaves ranks 'idle by construction', and I read the step-5 np=8 measurement the same way, reporting 'idle_ranks 6 of 8' as 75 percent of the machine idle. That is wrong. The counter is per-LEVEL: a rank holding no level-L block still owns level-0 work (level 0 covers every rank) and may hold blocks at other levels; even the TOTAL line means only 'owns no fine block'. So the number does not explain the 18 percent parallel efficiency - that still needs measured per-rank time, which is the point the metric section already makes. Also mark step 5 landed while stating that its m_rank_timing half, the half that measures idleness rather than inferring it, is not done.

Removing the multi-level subcycle PROHIBIT left twelve lines of explanation sitting in a list of checks with nothing after them - a comment justifying why something is NOT checked, in the file whose job is checking. The content is already in the two places it belongs: s_amr_advance_children's docstring (the level-wide walk and the interposed halo) and the C45DBB52 test comment (the amr_cpat_mar margin argument).

be94db3 shipped the per-level cap saying it was verified by reading the slot arithmetic, not by running, because no golden covers amr_max_level >= 3. It is now verified by running, and the evidence belongs where the next reader will be standing. m = 255, amr_max_level = 3, amr_buf = 48, np = 1: the fixed /2 keeps ONE oversized level-3 box where the per-level cap keeps 2, and the run aborts in s_amr_free_slot with a Fortran 'Invalid descriptor' and a core dump; the per-level cap exits 0. Two earlier counterfactuals produced byte-identical output and proved nothing - m = 63 was too small, and scaling the grid without scaling amr_buf left the boxes tracking the feature rather than the domain, far below either cap. A635AA56's comment already says it is the BUFFER that widens the box; applying that is what made the cap bind.

…nchmarking

The balancer only becomes observable when a level holds many boxes, and stripe cases collapse to ~2 boxes per level by np=8 - at which point every owner mapping produces an identical assignment. Scattered blobs give many disjoint tag clusters, hence many boxes per level. Hardcoded rather than an analytic patch on purpose: analytic expressions compile into case.fpp, so every variation costs a rebuild, whereas a(2:5) (count, seed, radius, amplitude) are read at run time and sweep freely against one binary. Centres come from an additive-irrational (Weyl) sequence, NOT random_number: each rank fills only its own cells, so the IC must be a pure function of position or ranks disagree at the subdomain seam and the decomposition stops being exact. Purely additive - a new case in a select, unreachable unless a case sets hcid = 299 - so no existing golden can enter it. Verified: all four targets build (the file compiles into pre_process via m_icpp_patches and simulation via m_ib_patches), pre_process runs it clean at 511x255 on 2 ranks. Also revises the plan doc: step 5 landed, acceptance criterion 2 promoted to a runnable step, and a retraction of this file's own earlier claim that m_rank_timing was unimplemented - it is fully wired at eight call sites.

…uption)

s_amr_regrid's 'brand-new region' branch - a level>=2 child whose region has no old fine data to cluster - emitted its box straight into boxes() without ever consulting amr_maxc_fit. It is the ONLY box-emitting path that skips s_amr_tile_box. The inset bounds the child as a FRACTION of its parent (span/4 each side), but the slot coord arrays are allocated ONCE to amr_ref_ratio*amr_maxc_fit, so the child must be bounded in ABSOLUTE cells. s_amr_build_block_coords then sizes fcb from the block's TRUE extent and writes past x_cb. Under -O2 that silently scribbles past a heap allocation every regrid and surfaces much later as 'corrupted size vs. prev_size' inside an unrelated free() - the crash site is nowhere near the bug site, which is why three passes of reading the seam/overlap/tiling code cleared every candidate. Integer division is the trigger: s_amr_tile_box splits a wide region into 63 and 64, not 64 and 64, and a parent of span 63 gives ins = 63/4 = 15 and a child of span 33 against a level-2 cap of 32, while span 64 gives exactly 32 and is fine. A one-cell difference in the parent flips it. Introduced by me in cfdd284 (step 4): s_amr_build_block_coords replaced a parent-relative computation with a global-extent one so a rank could build coords for a block whose parent it does not own, and the destination sizing did not follow. VERIFIED: bounds-checked CPU build (gfortran --no-gpu --debug, the ONLY configuration with -fcheck - cmake/GPU.cmake gives LLVMFlang none) traps at m_amr.fpp:584 'Index 128 of dimension 1 of array fcb above upper bound of 127' without this change, and with it the span-33 box becomes 17+16 and the run exits 0.

amr_max_grid_size had ZERO test coverage - not one golden or example case set it, only the schema, validator and definitions - despite being the mechanism the rank-independent-cap work rests on. Any defect reachable only through the pinned-cap path was invisible to CI, and one was: the brand-new-region box emitter skipped s_amr_tile_box and corrupted the heap (cfbaceb). VERIFIED to fail without that fix: 8 over-cap boxes and, under a bounds-checked build, 'Index 32 of dimension 1 of array fcb above upper bound of 31' (the trap index tracks the cap, max_f1 = 2*amr_max_grid_size - 1). Both knobs are load-bearing and narrow. amr_max_grid_size = 16 makes s_amr_tile_box split a wide region into 16 and 15, and span 15 gives ins = 15/4 = 3 and a child of 9 against a level-2 cap of 8; INTEGER DIVISION is the trigger, so a cap of 20 splits evenly (20/20 -> child 10, cap 10) and does not reproduce. The blob IC is equally load-bearing - a Sod-like patch grows level-2 regions that already have fine data, so the brand-new-region branch never fires and the bug is unreachable however the cap is set; three separate small 1D Sod candidates passed for exactly that reason. Also records the caveat in the plan doc: step 2's measured speedup was collected on this path, so it is no longer a clean measurement. 72/72 AMR goldens, precheck 7/7.

Two additions, both from today's heap-corruption hunt (cfbaceb). (1) s_amr_check_box_caps aborts if any level>=1 box exceeds amr_maxc_fit/amr_ref_ratio**(lev-1), checked once where the box set is final rather than trusting each emitter to have routed through s_amr_tile_box. A violation is otherwise silent and catastrophic - the coord arrays are sized to the cap, so an over-cap box writes past x_cb and corrupts the heap every regrid, surfacing much later inside an unrelated free(). VERIFIED BOTH WAYS: on the unfixed reproducer it prints '[amr] box cap violated: level 2 dim 1 span 9 > cap 8' and aborts by name instead of corrupting memory; and 71/71 existing AMR cases pass with it compiled in, zero false aborts. The false-positive hazard was real - in 1D/2D amr_maxc_fit(2:3) is 1, so the level-2 cap divides to 0 and a naive check would abort every 1D golden; it mirrors s_amr_tile_box's own max(tc,1) floor. (2) [amr-balance] now prints boxes_max/mean beside the weight imbalance, because measured per-rank time tracks BOX COUNT rather than cells: at np=8 the cell weight sat at 1.050 while box count was 1.308 and rank_time 1.259. Printing both makes that falsifiable in one run. 72/72 AMR goldens, precheck 7/7.

…ceiling

… tail

… block"

This reverts commit 00467d9.

…y block"

This reverts commit 9fd9592.

…ntier CCE gpu-acc)

A declare-create on a module allocatable binds a present-table entry to the descriptor at program init; the move_alloc at the allocation site then swaps that descriptor out, so every later kernel lookup misses. All 36 AMR tests on the CCE gpu-acc lane died with 'find_in_present_table failed' for amr_cons_br (432 hits) and amr_scr_prim (9). amr_cg has always used move_alloc + GPU_ENTER_DATA with no declare and works on every lane; this makes the other two match. The declares were added for the lib-4425 uninitialized-descriptor abort, but the move_alloc is what actually fixed that, so they were redundant as well as harmful. amdflang's OpenMP runtime creates the mapping implicitly, which is why no local gate could see this.

s_populate_variables_buffers sits inside s_compute_rhs, which PH_COARSE wraps whole, so a uniform (amr=F) run reported 95% 'coarse' and no communication at all. That made the one comparison that matters impossible: how much of AMR's ~31% communication share is AMR's own versus the solver's baseline.

Halo probe counts distinct block slots this rank's metadata reads touch between regrids, reduced with MPI_MAX over ranks rather than reported from rank 0 (which owns a domain corner and is the least connected rank). The reduction is entered by EVERY rank: an earlier draft placed it inside the proc_rank == 0 guard, so one rank called ALLREDUCE while the rest ran ahead into other collectives and the job died with MPI_ERR_TRUNCATE. Grid efficiency reports tagged versus covered coarse cells, which prices amr_tag_eps and amr_buf and bounds the quoted payoff.

…reshold

A histogram of the tagging ratio g/(2*r0) over ~49 regrids of this case shows the distribution is continuous from 0 to 0.5 with no gap, and amr_tag_eps = 0.05 already sits at a local minimum, so there is nowhere better to move it. Near the threshold the density is ~9,400 cells per unit ratio per regrid, so one flip across 49 regrids needs a perturbation of ~2e-6; roundoff is 1e-12 to 1e-14 and cannot reach that. The flipped tag is therefore a consequence of divergence, not its cause: roundoff is amplified by a chaotic configuration until it reaches ~1e-6, and only then do tags flip. t_step_stop 100 -> 25 keeps 12 regrids and the store still grows 25 -> 36 -> 49 blocks with the cap never binding, while giving roundoff far less room to grow. override_tol = 1e-11 covers the residual toolchain roundoff (CCE measured 1.06e-12) without being loose enough to hide a mesh divergence.

The allocation is guarded by hypo_nc_mode == hypo_nc_mode_dual_pass alone, but the teardown sat nested inside if (.not. igr), so any configuration with igr and dual_pass together leaked flux_n, flux_src_n and flux_gsrc_n along with their %sf payloads and device maps. Nothing couples igr to hypo_nc_mode in the validator, so the combination is reachable. Hoisted the teardown block out of the igr guard so the two are textually identical.

Nothing in the tree writes this array, but s_finalize_riemann_solver_hatR copies all of 1:sys_size out of it into flux_gsrc_n(id), which m_rhs folds into the RHS. Under cyl_coord with the dual-pass path that fed uninitialized memory into the solution. Mirrors the zeroing its twin flux_gsrc_rsx_vf already gets for the same reason.

s_compute_load_weight reads it on the first s_write_data_files, which for a run saving at t_step_start precedes any relaxation sweep, so the first load-weight diagnostic and load_weight.*.dat came from uninitialized memory. The device copy is the one that matters: the reader is a GPU_PARALLEL_LOOP and the only writer sits inside a device region, so a host-only assignment would have reached neither.

s_write_amr_restart marks format v2 with a NEGATIVE rank count, and the simulation reader was taught both formats, but the post_process reader was not. Its rank-count check therefore fires unconditionally (a negative can never equal num_procs) and reports a rank-count mismatch even when the counts match, which sends the reader after the wrong problem; it also still reads the 3*num_procs extents record that v2 does not write. Detect the negative marker in both the serial and MPI branches and abort saying what is actually wrong. This is a stopgap: reading v2 in post needs the whole-block ownership layout (one contiguous chunk per block, ownership from the file's 4-int record rather than derived geometrically), which is a separate change. Nothing in the default suite runs AMR with post_process as a target, which is why the break shipped unnoticed.

Only the parallel_io branch of s_write_amr_restart writes the negative rank count that marks format v2; the serial branch still writes a positive num_procs to p_all/p<rank>/<t_step>/amr_fine.dat. The guard added to the serial reader could therefore never fire, and its comment claimed a failure mode that path does not have. Verified against a real 8-rank AMR run: the parallel file's header is (-8, 224, 6), so the parallel guard is the one that matters.

The simulation path raises MFCException on a non-zero return code; the --test-all post_process path captured stdout and ignored the result entirely (common.system runs with check=False). post_process could abort, segfault or fail outright and the test still reported PASS as long as the simulation goldens matched. That is how a total break of the AMR post-process reader shipped unnoticed: --test-all ran post_process on every AMR case and threw the answer away. Verified both directions: an AMR test now fails where it previously passed, and non-AMR cases are unaffected.

Two defects, both in the AMR fine-block reader, both invisible until the test suite started checking post_process's exit code.

First: format v2 was never taught to post. The writer marks v2 with a negative rank count, the simulation reader handles both formats, and post did not, so its rank-count check fired unconditionally (a negative can never equal num_procs) and blamed a rank-count mismatch on runs that used the right number of ranks. v2 stores one contiguous chunk per block written by that block's single owner, plus a 4-int (owner + 1, m, n, p) record. Ownership now comes from that record rather than from the reader's own geometric intersection, which also fixes a case where a rank overlapping a block it does not own would read a -1 extent and abort. The whole chunk is read and this rank's intersection sub-box copied out, so s_setup_amr_block still reconstructs coordinates from a valid local coarse index. v1 files stay readable.

Second: post runs with a LARGER sys_size than the simulation for 5eq Lagrange bubbles, because it appends beta_idx as a post-only output slot. Comparing the file's conserved-variable count against post's inflated sys_size rejected every valid AMR + Lagrange-bubbles file. Size and read by the count the writer actually used.

[amr-grideff] printed amr_n_tagged, which counts THIS rank's local tag_grid, next to amr_n_covered, which comes from the global accepted box list (s_amr_cluster runs with reduce = .true., so every rank clusters the same global tag set and already holds the same volume). Printing the unreduced pair from rank 0 made the ratio wrong by about num_procs. Sum the numerator across ranks and leave the denominator alone. Every rank enters the collective; only rank 0 prints. Still approximate and marked as such in the source: the numerator mixes the level-1 and level-2 index spaces, and it is accumulated before the amr_buf pad and the box merge, so real over-coverage is worse than the printed ratio.

With l0_ntile > 0 the L0 tiles occupy slots 1..l0_slot_off of the SAME pool as the fine blocks and amr_num_blocks counts them, so s_write_amr_restart emits level-0 tile records into the AMR fine-block file. All three reader paths rejected them twice over: lvl == 0 fails the lvl < 1 test, and a tile's refinement ratio of 1 fails rr < 2. Every AMR + L0-tiles case therefore died in post_process with 'malformed fine-block header (level/extent inconsistent); the writer and reader header layouts have drifted' -- a message that blamed a drift which had not happened. A tile's data is the base grid re-tiled and is already in the level-0 restart file, so the overlay skips it while still advancing the file offset past the record. Reproduced and fixed on a CPU build, which is where CI sees it: all nine cases fail with the old reader and pass with the new one. They pass on amdflang GPU either way, so no local GPU gate could have caught this.

amr_n_shaped counts the coarse volume the FINAL boxes occupy, after s_amr_regrid_shape_boxes has padded by amr_buf, clamped, size-capped, clipped and tiled. amr_n_covered is taken before all of that, so it cannot see the pad, which is exactly where amr_buf acts: across an amr_buf 4 -> 1 sweep covered moved 0.4% while the refined volume moved 9%. [amr-grideff] now prints tagged, covered and shaped together. Also warn when the global accepted-box count exceeds amr_max_blocks: boxes contributed by other ranks were discarded silently, so the refined mesh stopped matching the tag field with no diagnostic. The existing capped flag covers only the tree bisection, not this truncation.

The runtime banner and the per-level distribution table both asserted 1.55x as a measured subcycle speedup. It is a phase-share model and was never measured. One matched-resolution arm pair (equal physical time, level 2 at the same dt in both) gave 2.84x, but that T arm's phase table omits rhs, seam, reflux, gather and rk entirely because the subcycle advance path carries no PH_* brackets, so it is a wall ratio with no accounting behind it. Both figures are now labelled for what they are. Separately, m_time_steppers said multi-level reflux under subcycling is future work; s_amr_advance_children has been calling s_amr_reflux_to_parent(dt_sub, .true.) for some time.

Records what the reviews established: the np8-512 weak-scaling ladder already existed and had never been analysed (rg:clus is O(P^3) and 75% of wall at np=512; rb:topo is exactly O(P^2)); the emitted box set is a single bounding box re-tiled, so amr_cluster_eff cannot affect it and the measured 2.20x over-coverage is this clusterer's best case on its most flattering geometry; and the per-step global scan work this plan had as item 1 is 2.5% of wall at np=512, measured, against a P^2 and a P^3 term beside it.

Under cfl_dt the simulation names each save by save_count = int(mytime/t_save) (m_start_up.fpp:798), so when adaptive dt grows enough for one step to cross TWO t_save boundaries the index skips and no file is written for the intervening value. The post loop walks indices 0..n_save-1 and the reader aborts on the first absent one, so every CFL-driven case died in post_process with 'File restart_data/lustre_N.dat is missing'. The IGR_triple_point case writes 60 saves at indices 0, 1, 3, 5, 7 and upward, and post demanded index 2. A skipped index is legitimate output, not a fault. f_save_exists lives beside the reader that owns the filename convention and only answers for the shared-file layout, since with file_per_process each rank holds a different file and the answer would not be rank-uniform; that path keeps the original fail-closed behaviour. Reproduced and fixed on a CPU build: the case fails before and passes after, and 12 non-CFL cases still pass.

…FlowCode#1795

4C751DAF aborts on the Frontier CCE cpu lane with 'No Lagrangian bubbles remain in the domain', and passes on amdflang GPU and gfortran CPU. Measured on gfortran over the full run, from the case's own lag_bubble_evol output: the single bubble moves 0.5 -> 0.5000076, its radius goes 0.008 -> 0.0079987 (it shrinks), and the void fraction in its cell peaks at 0.0335 against a valmaxvoid threshold of 0.99 -- a 3.1x margin. Neither removal criterion is reachable, so the count reaching zero on CCE is a symptom of the bubble state degrading there, not a physical outcome. The case is therefore NOT loosened to dodge the abort: with that much margin on working toolchains, any such change would hide whatever actually goes wrong on CCE. Only the --test-all re-run trips it, so the primary run and the golden comparison still cover the case fully. This is present on master too, invisible because the suite discarded the --test-all exit code.